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Abstract_ 

Stakeholders in STEM education have called for integrating engineering content 
knowledge into STEM-content classrooms. To answer the call, stakeholders in 
science education announced a new framework. Next Generation Science 
Standards, which focuses on the integration of science and engineering in K-12 
science education. However, research indicates many science teachers, 
particularly those traditionally prepared to teach within a specific science content 
domain, need to broaden their knowledge in engineering content areas for 
successful integration. In this regard, researchers have suggested that new 
integrated STEM curricula should contain a list of key concepts for 
understanding the specific engineering content area. Therefore, there is a need 
for generating key concepts in critical engineering areas enabling science 
teachers to implement engineering into science classrooms. Using a modified 
Delphi research design, we identified and verified key concepts in earthquake 
engineering necessary for high school learners to acquire a basic understanding 
of earthquake engineering. As a result, we created a key concepts list and strand 
map with 35 key earthquake-engineering concepts. High school science teachers 
as well as other teachers in STEM content areas can use these key concepts to 
understand and teach earthquake engineering content in their classrooms. 


Introduction 

Science, Technology, Engineering, and Mathematics (STEM) education in the last two decades has been a 
critical focus for stakeholders in public education (Erdogan & Stuessy, 2015a, 2015b; Lopez et ah, 2011; 
Nathan, Atwood, Prevost, Phelps, & Tran, 2011; Navruz, Erdogan, Bicer, Capraro, & Capraro, 2014; Wilson, 
2011). Policymakers, researchers, and educators recognize the role of STEM education on the economic welfare 
and leadership status of the US (President’s Council of Advisors on Science and Technology, 2010), as well as 
students’ development of science literacy (Duschl, Schweingruber, & Shouse, 2007). Recent reports from 
leading stakeholders (i.e.. Next Generation Science Standards [NGSS], 2013; National Research Council 
[NRC], 2014b), furthermore, have stressed the need for improving and expanding STEM education and 
enhancing student readiness for future careers reliant on STEM content knowledge (NRC, 2012). In addition, 
new guidelines for K-12 science and engineering education stress STEM integration (NRC, 2014b) connecting 
science, technology, mathematics, and engineering content among, rather than within individual domains. As 
engineering knowledge “utilizes concepts in science and mathematics as well as technology tools” (NRC, 
2014b, p. 14), some leading researchers have identified engineering as the likely catalyst for the integrating all 
STEM areas (Katehi, Pearson, & Feder, 2009). As a result, stakeholders in STEM education have renewed the 
call to integrate engineering content knowledge into science, mathematics, and technology classrooms. 

Teachers have been encouraged to teach many STEM-related content areas through the integration of 
engineering rather than focusing on the specific content area. For example, the authors of the Common Core 
State Standards for Mathematics (Common Core State Standards Initiative, 2011) announced the need for 
integrating mathematics with science and engineering. Similarly, the International Technology and Engineering 
Educators Association stressed the necessity of understanding connections across science, technology, 
engineering, and mathematics (International Technology Education Association, 2007). Moreover, important 
policy documents in science education, such as Benchmarks for Science Literacy (American Association for the 
Advancement of Science [AAAS], 2009), have announced the need for the integration of all STEM content 
areas, including engineering. Currently, however, most science teachers still restrict their content knowledge 
preparation to one specific content area (e.g., life science, chemistry, physics, and earth science) rather than 
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broadening their knowledge to include engineering content areas that would facilitate successful integration. 
Although the need for integrating engineering to address the purpose of total STEM integration within the other 
three STEM content areas (i.e., science, mathematics, technology) has been stated, only recently and with 
limited implementation has engineering been integrated into science classrooms. Knowing this, NGSS (2013) 
announced a new framework with strong implications for enhancing STEM education. Specifically, this 
framework focuses on the integration of science and engineering (S&E) in K-12 science education. A majority 
of US states have already proposed implementing this framework into their science curriculum (NRC, 2014b). 


NGSS Framework 

The NGSS framework (NGSS, 2013), based on A Framework for K-12 Science Education: Practices, 
Crosscutting Concepts, and Core Ideas (NRC, 2012), called for deeper connections among STEM subjects 
(NGSS, 2013; NRC, 2014b). The framework was outlined around concepts for K-12 science education derived 
from existing documents including the National Science Education Standards (NRC, 1996), the Benchmarks for 
Science Literacy (AAAS, 2009), the Atlas of Science Literacy (AAAS, 2001), the Science Framework for the 
2009 National Assessment of Educational Progress (National Assessment of Educational Progress, 2009), and 
the Science College Board Standards for College Success (College Board, 2009). Consequently, the NGSS 
framework reflects previous standards considered crucial for successful K-12 science education. The goal of the 
framework is as follows: 

To ensure that by the end of 12th grade, all students have some appreciation of the beauty and wonder 
of science; possess sufficient knowledge of science and engineering to engage in public discussions on 
related issues; are careful consumers of scientific and technological information related to their 
everyday lives; are able to continue to learn about science outside school; and have the skills to enter 
careers of their choice, including (but not limited to) careers in science, engineering, and technology. 
(NRC, 2012, p. 1) 


Table 1. The three dimensions of the NGSS framework 



Asking questions (for science) and defining problems (for engineering) 
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Developing and using models 
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Planning and carrying out investigations 
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Analyzing and interpreting data 
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Using mathematics and computational thinking 
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Constructing explanations (for science) and designing solutions (for engineering) 


Engaging in argument from evidence 


Obtaining, evaluating, and communicating information 
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Cause and effect: Mechanism and explanation 
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Scale, proportion, and quantity 
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Systems and system models 
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Energy and matter: Flows, cycles, and conservation 

V 

Structure and function 


Stability and change 


Physical Sciences (PS) 1: Matter and its interactions 


PS2: Motion and stability: Forces and interactions 
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PS3: Energy 

TJ 

PS4: Waves and their applications in technologies for information transfer 

QJ 

U 

Life Sciences (LS) 1: From molecules to organisms: Structures and processes 

O 

U 

LS2: Ecosystems: Interactions, energy, and dynamics 


LS3: Heredity: Inheritance and variation of traits 

a 

C 

LS4: Biological evolution: Unity and diversity 

[S- 

Earth and Space Sciences (ESS) 1: Earth’s place in the universe 

*3 

in 

ESS2: Earth’s systems 

5 

ESS3: Earth and human activity 


Engineering, Technology, and Applications of Science (ETS)l: Engineering design 


ETS2: Links among engineering, technology, science, and society 
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For successful K-12 science education, the NGSS framework outlines three dimensions: (a) Scientific and 
Engineering Practices, (b) Crosscutting Concepts, and (c) Disciplinary Core Ideas (Table 1). The NGSS 
framework stresses meaningful learning in S&E through the integration of the three dimensions into standards, 
curriculum, instruction, and assessment (NRC, 2012). The first dimension. Scientific and Engineering Practices, 
emphasizes the essential role of practices for student learning of S&E in K-12 science classrooms. Mastering 
these practices helps students see similarities and differences between science and engineering. In addition, this 
dimension allows students to establish a better understanding of how scientific knowledge and engineering 
solutions are developed (NRC, 2012). The second dimension, Crosscutting Concepts, highlights critical 
concepts that “provide students with an organizational framework for connecting knowledge from the various 
disciplines into a coherent and scientifically based view of the world" (NRC, 2012, p. 83). Familiarity with 
these concepts in K-12 science classrooms supports student understanding of disciplines within S&E while 
providing a method to access information across these disciplines. The third dimension. Disciplinary Core 
Ideas, outlines core ideas for the focus of S&E education in K-12 science classrooms. Mastering these core 
ideas through learning progressions (Duschl et al., 2007) allows students to continually learn core ideas within 
S&E and develop deep understanding of multiple topics. This dimension, therefore, allows more time for 
teachers to teach and students to learn each topic over the course of students’ K-12 science education. The three 
dimensions within the NGSS framework satisfy the overall goal of STEM integration by implementing 
engineering content knowledge in K-12 science classrooms. However, additional challenges for successful 
STEM education must also be considered before successful integration can occur. 


Identifying Knowledge Bases in Targeted Engineering Areas 

Developing an understanding of engineering pedagogical content knowledge can be difficult for many K-12 
science teachers. For successful STEM learning, teachers must recognize engineering knowledge with a 
consideration for student learning levels (e.g., elementary, middle, or high school). In doing so, identifying 
knowledge bases (i.e., key concepts identification; see Rossouw, Hacker, & de Vries, 2011; Wicklein, Smith, & 
Kim, 2009; Wooten, Rayfield, & Moore, 2013) for the targeted engineering knowledge becomes critical in 
facilitating student comprehension of the concepts associated with a level of understanding at an appropriate 
level. In addition, the identification of key concepts is essential for teachers to draft well-defined learning 
objectives, plan suitable teaching strategies, and create meaningful assessment strategies for measuring student 
understanding. Some researchers have noted a “fear of engineering” in STEM teachers, due to the complexity of 
engineering content knowledge (NRC, 2014b). Such a fear can result in a teacher’s lack of confidence in 
teaching engineering (Banilower et al., 2013). The prior identification of key concepts, therefore, has the 
potential to reduce the complexity of engineering content for STEM teachers while also increasing their 
understanding and confidence in teaching engineering. 

Several researchers have noted that the identification of key concepts is important for developing valid and 
reliable assessment strategies (e.g., Darmofal, Soderholm, & Brodeur, 2002; Walshe, 2007), especially within 
the complex domains of engineering knowledge (Cavlazoglu & Stuessy, 2016; Darmofal et al., 2002; Martinez, 
Perez, Suero, & Pardo, 2013; Wilson, 2011). Key concepts that have been previously identified can be used to 
develop measures for assessing knowledge. When key concepts have not been previously identified, the 
development of assessments becomes more difficult. Teachers must conduct their own research to identify key 
concepts in targeted engineering content areas. For example, Rossouw, Hacker, and de Vries (2011) conducted a 
Delphi study to identify key concepts in engineering and technology education. Similarly, Wooten et al. (2013) 
also used a Delphi study to identify 21 STEM concepts associated with a junior livestock project. Osborne, 
Ratcliffe, Collins, Millar, and Duschl (2003) also conducted a three-stage Delphi study to identify key concepts 
in the nature of science to provide students with a better understanding of the topic. Similarly, the NGSS (2013) 
research team conducted an extensive analysis of related documents (e.g.. National Science Education 
Standards, Benchmarks for Science Literacy, the Atlas, Science Framework for the 2009 National Assessment 
of Educational Progress, and Science College Board Standards for College Success) to identify the crosscutting 
concepts in the NGSS framework in science education. After the release of this framework, researchers from the 
National Association for Research in Science Teaching (NARST) suggested ways to develop new engineering 
lesson plans, which would include identified key concepts in the targeted engineering content area (Purzer, 
Moore, Baker, & Berland, 2014). This research group recognized key concepts as an essential element of the 
curriculum enabling science teachers to implement engineering into science classrooms. At this time, however, 
efforts to identify key concepts for teaching engineering in K-12 science education are still limited. The 
earthquake engineering education professional development team at Texas A&M University learned this first 
hand as they attempted to develop a curriculum for science teachers who would be attending a summer 
professional development experience about earthquake engineering. 
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Earthquake Engineering as a Critical Content Area in Science Classrooms 

The recent NGSS framework in science education suggests integrating critical engineering content areas into 
science classrooms (NGSS, 2013) and finding appropriate engineering areas that allow the implementation of 
the three dimensions of the NGSS (see Table 1). Earthquake engineering fulfills the definition of a critical 
engineering content area in that the content domains of earthquake engineering cover most of the Disciplinary 
Core Ideas defined in the NGSS framework. Specifically, the three disciplinary core ideas (i.e., physical 
sciences; earth and space sciences; and engineering, technology, and applications of science) can be taught 
through earthquake engineering implementation. These three disciplinary core ideas represent approximately 
75% of the disciplinary core ideas that have been a focus in the NGSS framework. 

Furthermore, earthquake engineering content has the potential to improve the literacy level of citizens about 
earthquake resilience (NRC, 2011b). In 2011, the National Hazards Reduction Program announced a need in 
earthquake engineering education research to achieve an earthquake-resilient society and suggested improving 
understanding of earthquake engineering processes and impacts (NRC, 2011a). The NRC organized a 
community workshop with 37 researchers and practitioners from a wide range of earthquake engineering 
disciplines to identify problems and high-priority research areas in earthquake engineering related research. This 
workshop revealed the need to focus on social systems as well as designed systems to improve community 
resilience in earthquake engineering research (NRC, 2011a). The researchers and practitioners in this workshop 
also noted the limited emphasis on social and designed systems in previous earthquake engineering research. As 
a result, stakeholders in earthquake engineering education had not yet achieved the goal of creating an 
earthquake-resilient society. Earthquake engineering has the potential to be a critical content area for NGSS’s 
recent call for significant engineering content (Cavlazoglu & Stuessy, 2016). Currently, however, the key 
concepts necessary for science teachers and students in K-12 education to learn in order to understand 
earthquake engineering have yet to be identified. In this study, we propose to identify and verify the key 
concepts in earthquake engineering necessary for high school science teachers and students to understand 
earthquake engineering. These concepts will help science teachers, particularly those who have been 
traditionally prepared to teach a specific science content domain, and their students to understand the 
multifaceted content domain of earthquake engineering. 


Method 

Type of Research Design 

In this study, we used a modified Delphi research design (Skulmoski, Hartman, & Krahn, 2007; see Figure 1). 
The purpose of the Delphi design is to obtain a consensus from a group of experts when there is insufficient 
knowledge about a phenomenon (Borg, Gall, & Gall, 2003; Wicklein et al., 2009). The Delphi design allows a 
group of experts to share thoughts, exchange aspects, and ultimately reach consensus about a phenomenon 
(Osborne et ah, 2003; Rossouw et ah, 2011; Skulmoski et ah, 2007; Wicklein et ah, 2009; Wooten et ah, 2013). 
Researchers using the Delphi design have indicated that this method is one of the best research strategies to 
ascertain a beginning knowledge base in topics that have no foundation in prior research (Delbeq, Van de Ven, 
& Gustafson, 1975; Skulmoski et ah, 2007; Wicklein et ah, 2009; Wooten et ah, 2013). In addition, the Delphi 
design can be modified based on the purpose of the study, availability and type of data, and number of experts in 
the researched area (Skulmoski et ah, 2007). When a sufficient number of experts is available (e.g., n>30), the 
classic Delphi design procedures with three rounds of communication can be used. If participants in a classic 
Delphi design do not have a consensus after three rounds, additional rounds can be added until a sufficient level 
of consensus is reached among participants. Furthermore, different research methods (i.e., qualitative, 
quantitative, and mixed) can be used in Delphi studies based on the research questions and availability of data 
type. When the number of experts is limited in the researched area, further verification with another sample of 
experts should occur (Skulmoski et ah, 2007). 

In this study, the number of experts in earthquake engineering education was limited. As a result, we used a 
modified Delphi research design, implementing a two-phase process to identify and verify the knowledge base 
for earthquake engineering at the high school level. The goal was to assist science teachers in our summer work 
program in acquiring a sufficient understanding of earthquake engineering for them to implement this content 
into their science classrooms. In addition, teachers can also use these concepts to develop strategies for 
assessing students’ level of understanding, following recommended assessment practices by the NGSS 
framework developers and other stakeholders in science education (e.g., NRC, 2014a; Purzer et ah, 2014). 
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Figure 1 displays the two phases of the modified Delphi study we employed in this study: The first phase- 
identification and the second phase- verification. The purpose of the first phase was to identify the essential key 
concepts in earthquake engineering for high school science teachers and students to learn. During this phase, the 
Earthquake Engineering Education Project (EEEP) researchers conducted intensive research from related 
literature in both science and earthquake engineering education. They participated in five panel meetings to 
deliberate, discuss, and negotiate the final list of key earthquake engineering concepts they felt were essential 
for high school teachers and students to know and understand. The purpose of the second phase was to verify 
the key concepts from the original list with a larger panel of experts from varying disciplines. We used a one- 
round Delphi study via an online questionnaire to verify key concepts from the original list. 


FIRST PHASE: Key Concepts Identification 


ONLINE QUESTIONNARIRE 


SECOND PHASE: Key Concepts Verification 


Figure 1. Modified Delphi research design used in this study 


Participants 

The participants in the two phases of the study were nine experts in earthquake engineering education research. 
We used purposive sampling to gather experts’ opinion (Skulmoski et al., 2007; Wooten et ah, 2013) in order to 
generalize a list of key concepts in earthquake engineering. In the first phase of the modified Delphi study, three 
researchers identified the original key concepts list. They conducted intensive research from related literature to 
identify potential key concepts and convened five panel meetings over six months to discuss, deliberate, defend, 
and make decisions regarding the inclusion of earthquake engineering concepts. These researchers included an 
associate professor in science education who holds a PhD in science education. Her research interests include 
broader impacts in science and engineering research, which led her to become an expert in developing effective 
STEM workshops for K-12science teachers. The second researcher holds a master’s degree in physics. Her 
research interests include social learning in teaching and learning science and earthquake engineering. The third 
researcher is a PhD student in science education who also holds a master’s degree in science education. His 
research interests include developing authentic teaching, learning, and assessment strategies in earthquake 
engineering education. All researchers in the first phase were members of the Earthquake Engineering 
Education Project (EEEP) workshop development team. 

The second phase of this Delphi study engaged six participants. These included three participants from science 
education and three participants from civil engineering with research interests and expertise in earthquake 
engineering. All participants were from tier-1 research universities in the US and met the four expertise criteria 
for Delphi studies as identified by Adler and Ziglio (1996): (1) sufficient knowledge and interest with the 
phenomena under investigation, (2) capacity and interest to participate, (3) available to spare sufficient time for 
participating and (4) efficient communication skills. 


Data Collection and Analysis Procedures 

For the first phase of the study, the data were collected using concept lists that the EEEP research team 
generated after each panel meeting. For the second phase of the study, the data were collected using an online 
questionnaire, and six experts were asked to verify the key concepts that had been identified in the first phase. 

In the first phase of the study, we used descriptive statistics to report each panel’s products resulting in a list of 
key concepts, using a criterion of 100% consensus for inclusion of the concept in the key concept list. In the 
second phase of the study, we again used descriptive statistics to report the results of respondents’ ratings, 
including means, medians, and modes for each concept. For this phase, we established a criterion level of 80% 
consensus as suggested by related Delphi study literature (e.g., Wooten et ah, 2013) to verify the inclusion of a 
concept in the key concepts list. 
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Procedures and Results 

This Delphi study consisted of two phases including first phase for key concepts identification and second phase 
for key concepts verification. 


First Phase: Key Concepts Identification 

This phase consisted of five face-to-face panel meetings of three researchers to identify and negotiate a final list 
of key concepts appropriate for learners at the high school level in developing a basic understanding of 
earthquake engineering. Each panel meeting focused on a particular phase of key concept identification, as 
follows: (1) Resource Document Identification, (2) Content Domain Identification, (3) Initial Key Concept 
Identification, (4) Key Concept List Completion, and (5) Key Concept List Confirmation. 


Panel 1: Resource Document Identification 

Three researchers convened to identify resource documents for identifying the key concepts appropriate for 
learners at the high school level in order to develop a basic understanding of earthquake engineering. 
Specifically, they were asked, “What are the important documents we need to use as references in identifying 
the key concepts necessary for high school learners to understand earthquake engineering?” The researchers 
discussed critical documents to use as resources, and all agreed on seven nationally published documents 
spanning a period of seventeen years (see Table 2). 


Table 2. Source documents for identifying key concepts in earthquake engineering 


Year 

Name 

Reference 

2013 

Next Generation Science Standards 

NGSS 

2011 

Grand Challenges in Earthquake Engineering Research: A Community 
Workshop Report 

NRC 

2011 

National Earthquake Resilience: Research, Implementation, and 
Outreach 

NRC 

2009 

Benchmarks for Science Literacy 

AAAS 

2007 

Atlas of Science literacy. Volume II 

AAAS 

2001 

Atlas of Science literacy. Volume I 

AAAS 

1996 

National Science Education Standards 

NRC 


At the end of the first panel meeting, researchers agreed that their next step would be to identify the domain 
areas in earthquake engineering. They also agreed to review the source documents on their own and bring ideas 
to discuss and finalize at the next panel meeting. The researchers scheduled the second panel for approximately 
two months later. 


Panel 2: Content Domain Identification 

As earthquake engineering is an interdisciplinary content area, critical domains related with science, as well as 
its STEM connections, needed to be identified. In this panel, researchers identified the content domain areas 
subsuming the key concepts to include in high school earthquake engineering. After spending two months 
reviewing the documents (see Table 2), the participants discussed the identified five critical domains in 
earthquake engineering, with 100% agreement. These domains were (1) Physical Systems, (2) Designed 
Systems, (3) Social Systems, (4) Earth Systems, and (5) STEM Proficiencies. At the end of the second panel 
meeting, participants agreed to identify key concepts representing the most essential ideas in earthquake 
engineering and to place them within each of the five critical domains areas. They scheduled the third panel 



46 


meeting one month later. Each participant referred to the documents again, this time for the purpose of 
identifying key concepts and placing them into the related domain areas. 


Panel 3: Initial Key Concept Identification 

In this panel, researchers were asked, “What domain-specific concepts are critical for high school science 
teachers and students to understand earthquake engineering?” Each researcher indicated concepts she/he found 
important in her/his individual review of decided literature (see Table 2) and discussed each concept in detail in 
the panel. At the end of this panel, the three researchers identified 23 key concepts (see Table 3), decided to 
continue to identify any remaining key concepts, and scheduled the fourth panel meeting one month later. 

Table 3. Identified key concepts list in panel 3 
Domain Area Key Concepts 


Physical Systems Force, Energy, Motion, Transfer 


Designed Systems 
Social Systems 
Earth System 


Efficacy, Cost, Safety, Constraints, Regulations, Risks, Resources 

Social Response, Urban Infrastructure, Decision Making, Governance, Policy, 
Finance 

Earthquakes, Geographic Landforms, Plate Boundaries 


STEM Proficiencies Observation, Measuring, Prediction 


Panel 4: Key Concept List Completion 

In this panel, participants completed the key concepts list by adding 12 more concepts to the list. The added 
concepts were as follows: “disturbance” and “waves” in the Physical Systems domain; “reliability” and 
“resilience” in the Designed Systems domain; “oversight” and “prevention” in the Social Systems domain; 
“epicenter” and “worldwide patterns” in the Earth System domain; and “mathematical modeling,” “system 
thinking,” “theorizing,” and “tools” in the STEM Proficiencies domain. At the end of this meeting, participants 
decided to meet in another panel to review the key concepts for a final time. The fifth panel was scheduled for 
three weeks later. 


Panel 5: Key Concept List Confirmation 

In this final panel meeting, all concepts were discussed and confirmed. In addition, two more concepts, 
“redundancy” and “trade-offs,” were added to the key concepts list. With two more concepts identified in this 
panel, the total came to 37 concepts distributed within the five domain areas (see Table 4). 


Table 4. Final version of identified key concepts list in panel 5 


Domain area 

Key concepts 


Physical Systems 

Force, Energy, Motion, Transfer, Disturbance, Waves 


Designed Systems 

Efficacy, Cost, Safety, Constraints, Regulations, Risks, Resources, 
Resilience, Trade-offs, Redundancy 

Reliability, 

Social Systems 

Social Response, Urban Infrastructure, Decision Making, Governance, Policy, 
Finance, Oversight, Prevention 

Earth System 

Earthquakes, Geographic Landforms, Plate Boundaries, Epicenter, 
Patterns 

Worldwide 

STEM Proficiencies 

Observation, Measuring, Prediction, Mathematical Modeling, System Thinking, 
Theorizing, Tools 
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Second Phase: Key Concepts Verification 

In this phase, six earthquake-engineering experts were asked to verify the identified 37 concepts by indicating 
their level of agreement on each concept. An online questionnaire was sent to the experts via email, which was 
completed in three weeks. 


Table 5. Results for key concepts verification 


Domain area 

Key concept 

Mean 

Mode 

Range 

% Rating with 4 or 5 

Physical Systems 

Force 

5.00 

5 

0 

100 


Energy 

5.00 

5 

0 

100 


Motion 

5.00 

5 

0 

100 


Waves 

5.00 

5 

0 

100 


Transfer 

4.83 

5 

1 

100 


Disturbance 

4.17 

5 

2 

67 

Designed Systems 

Cost 

4.83 

5 

1 

100 


Safety 

4.83 

5 

1 

100 


Risks 

4.83 

5 

1 

100 


Constraints 

4.67 

5 

1 

100 


Regulations 

4.50 

5 

1 

100 


Resources 

4.50 

5 

1 

100 


Resilience 

4.50 

5 

1 

100 


Efficacy 

4.33 

5 

2 

83 


Trade-offs 

4.33 

5 

2 

83 


Redundancy 

4.33 

5 

2 

83 


Reliability 

4.33 

5 

2 

67 

Social Systems 

Urban Infrastructure 

4.67 

5 

1 

100 


Governance 

4.50 

5 

1 

100 


Finance 

4.50 

5 

1 

100 


Policy 

4.33 

4 

1 

100 


Social Response 

4.33 

4 

1 

100 


Decision Making 

4.33 

5 

2 

83 


Prevention 

4.00 

5 

2 

50 


Oversight 

3.67 

4 

2 

67 

Earth Systems 

Epicenter 

5.00 

5 

0 

100 


Earthquakes 

4.83 

5 

1 

100 


Plate Boundaries 

4.83 

5 

1 

100 


Geographic Landforms 

4.50 

5 

1 

100 


Worldwide Patterns 

4.33 

5 

2 

83 

STEM 

Proficiencies 

Mathematical Modeling 

5.00 

5 

0 

100 


Observation 

4.50 

5 

1 

100 


Measuring 

4.50 

5 

2 

83 


Prediction 

4.50 

5 

2 

83 


System Thinking 

4.50 

5 

2 

83 


Tools 

4.17 

5 

3 

83 


Theorizing 

3.33 

3 

1 

33 


The questionnaire provided a brief summary of the previous concept identification process as well as rationale 
of the study. The 37 identified concepts were presented within the five domain areas: Physical Systems (six 
concepts); Designed Systems (eleven concepts); Social Systems (eight concepts); Earth Systems (five concepts), 
and STEM Proficiencies (seven concepts). Each of the six experts indicated their level of agreement for each 
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concept to complete the verification process. Participant responses to the online questionnaire were analyzed 
and results for each concept are shown in Table 5. 

In Table 5, the first column indicates the domain areas for key concepts. The second column lists the concepts 
identified by experts in the phase one. The third and fourth columns contain measures of center, specifically the 
mean and mode values for experts’ responses on the questionnaire. The fifth column contains a measure of 
spread, namely the range or difference between the highest and lowest response values. The sixth column 
contains a measure for the shape of experts’ responses. 


The concept verification process resulted in 35 concepts verified by six experts. These 35 concepts reached the 
consensus criterion (i.e., 80% agreement; Wooten et al., 2013) with a minimum mean of 4.00. Only two 
concepts, “oversight” and "theorizing,” had a lower mean of 4.00 (i.e., M oversight =3.67 and M, heorizing = 3.33). 
These concepts were dropped from the final key concepts list to yield a final list of 35 identified and verified 
concepts. Table 6 lists the 35 concepts considered essential for high school learners to understand earthquake 
engineering. 


Table 6. Final version of key concepts list 


Domain area 

Key concepts 


Physical Systems 

Force, Energy, Motion, Transfer, Disturbance, Waves 


Designed Systems 

Efficacy, Cost, Safety, Constraints, Regulations, Risks, Resources, 
Resilience, Trade-offs, Redundancy 

Reliability, 

Social Systems 

Social Response, Urban Infrastructure, Decision Making, Governance, Policy, 
Finance, Prevention 

Earth System 

Earthquakes, Geographic Landforms, Plate Boundaries, Epicenter, 
Patterns 

Worldwide 

STEM Proficiencies 

Observation, Measuring, Prediction, Mathematical Modeling, System Thinking, 
Tools 


Furthermore, we created a strand map (see Figure 2) to facilitate high school teachers’ use of the key concepts in 
teaching earthquake engineering. The strand map illustrates the key concepts in a visual form showing 
relationships among and across concepts in the five domains of earthquake engineering. The map indicates 
relationships among concepts within a single strand (all of the same color, unbroken lines) and across strands 
(see dotted lines). The strand map follows conventions established by the AAAS (AAAS, 2001; 2007), in which 
they depict connections between and among strands for four grade level bands. Our map, in contrast, indicates 
relationships between and among concepts within the strands for only high school (grades 9-12) learners. As 
with conventions established by Novak (2010), concepts are arranged hierarchically, arrows indicate the 
direction and connecting words indicate the nature of the relationship between the connected concepts. 


Conclusion 

New K-12 STEM education guidelines emphasize integrating engineering knowledge in STEM-content 
classrooms, with engineering knowledge serving as a catalyst for the integration of STEM content areas. In 
addition, the recent NGSS framework (2013) stresses integration of science and engineering in K-12 science 
classrooms for successful STEM education. This framework has been purposed by a majority of US states to be 
implemented into their science curricula (NRC, 2014b). However, research continues to show that science 
teachers still have a “fear of engineering” because of their limited engineering content knowledge. Furthermore, 
most of these teachers do not have access to well-defined knowledge bases (e.g., key concepts) in critical 
engineering content areas. Currently, defined engineering knowledge bases at the high school level do not exist 
(NRC, 2014b). In this regard, researchers have suggested that new integrated STEM curricula contain a list of 
key concepts critical in understanding the specific engineering content area (e.g., earthquake engineering; 
Cavlazoglu & Stuessy, 2016). The purpose of this modified Delphi study was to identify and verify key 
concepts in earthquake engineering necessary for high school learners to acquire a basic understanding of 
earthquake engineering as a human activity integrating content and procedural knowledge, expressed as 
“content domains.” In a two-stage process, (1) three researchers in earthquake engineering education identified 
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Figure 2. Strand-map linking concepts identified within the five domain areas (i.e., “strands”) of earthquake 
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37 key concepts in five domains with 100% consensus, and (2) six experts in science education and civil 
engineering with research interests and expertise in earthquake engineering verified 35 of these concepts with at 
least 80% consensus. A key concepts list and strand map with 35 earthquake engineering key concepts were 
created to support high school students’ development of an understanding about earthquake engineering. High 
school science teachers as well as other teachers in STEM content areas (i.e., mathematics, technology, and 
engineering) can use these key concepts to understand and teach earthquake engineering content in their STEM 
classrooms. 


Implications 

At least four implications exist in the results of this modified Delphi study. First, high school STEM teachers 
can use the engineering key concepts list for understanding and teaching earthquake engineering in their STEM 
classrooms. As research suggests, it is crucial to identify key concepts in critical engineering content areas for 
STEM teachers’ better understanding and teaching of engineering content (Purzer et ah, 2014; Rossouw et ah, 
2011; Wicklein et ah, 2009; Wooten et ah, 2013). STEM teachers can use the key concepts list to better 
understand and implement earthquake engineering (i.e., a critical engineering content area) into their 
classrooms. The strand map can help teachers see the key concepts in a visual form illustrating relationships 
among concepts within a single domain and across domains. 

Second, this modified Delphi study can be a model for others to identify and verify key concepts in other 
engineering content areas. Stakeholders in STEM education suggest identification of key concepts in critical 
engineering content areas for high school STEM teachers to increase their engineering content knowledge and 
teach the engineering content confidently without a “fear of engineering” (NRC, 2014) due to the complexity of 
engineering content. The identifications of key concepts can make complex and interdisciplinary engineering 
content areas more implementable for STEM classrooms. 

Third, curriculum developers in science, or STEM education, can benefit from using key concepts in developing 
engineering integrated curricula for successful STEM education (Rossouw et ah, 2011). The list can provide a 
reference for developing lesson plans, learning activities (Purzer et ah, 2014), and assessments to measure 
complex and interdisciplinary engineering content areas (Cavlazoglu, 2015; Cavlazoglu & Stuessy, 2016; NRC, 
2014a). As identified, key concepts in the list are related to nearly 75% of the disciplinary core ideas purposed 
in the NGSS framework. Implementing these earthquake-engineering concepts into science classrooms can be 
beneficial in the integration of the NGSS framework (NGSS, 2013) into science classrooms. 

Finally, the implementation of earthquake engineering content can increase students’ literacy level about 
earthquake resilience. Stakeholders in earthquake engineering research (e.g., NRC, 2011a, 2011b) identified the 
need to focus on social systems and designed systems domains in teaching and learning earthquake engineering 
to improve citizens’ literacy level about earthquake resilience. Identified social and designed systems key 
concepts in this modified Delphi study, therefore, can be useful to develop a better understanding of these two 
earthquake engineering domains and result in achieving an earthquake-resilient society. 
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